Multiple Variant (A/B test)
Introduction
A/B Testing (also known as split testing) is a method for comparing two or more variations of content to determine which performs better. In the Chisquares platform, the A/B Test question type allows users to randomly present different question variations to respondents, ensuring unbiased data collection. This feature is particularly useful for optimizing survey design, assessing content effectiveness, and evaluating user engagement.
This manual provides a comprehensive guide to setting up and managing A/B Test questions within Chisquares, covering basic setup, advanced features, and step-by-step instructions.
Features
- Question Text Input: Create and format the core question prompt.
- Response Options: Define selectable answers manually or dynamically.
- Required Toggle: Make the question mandatory to proceed.
- Random Assignment: Each participant is randomly assigned to view one media file (image, video, or audio) before answering a question.
- Media Upload: Users can upload multiple media files to be randomized among participants.
- Foundation Question Support: Compatible with foundational question types such as Single Value Text, Likert Scale, Multiple Choice (MCQ), Sliding Scale, Yes/no, Multiple Responses, Dropdown, and Prompts.
Use Cases
The A/B Test (Multiple Variant) Question type is designed to help researchers compare the performance of two or more question variations. By randomly (or conditionally) assigning different versions of a question to respondents, researchers can determine which version is more effective, engaging, or persuasive—without bias.
Common use cases include:
-
Message Testing: Evaluate different versions of marketing or policy messages.
Example: “Which headline better communicates the urgency of climate action?” -
Design Comparison: Test reactions to alternate layouts, packaging, or imagery.
Example: “Compare user responses to different versions of product labels.” -
Question Format Optimization: Assess which question format (e.g., Likert vs. MCQ) yields better response quality.
Example: “Determine whether scale-based or categorical framing leads to more thoughtful responses.” -
Behavioral Research: Explore how slight wording differences affect perceptions or decisions.
Example: “Does rephrasing the question change how harmful respondents perceive a product to be?” -
Content Personalization Trials: Experiment with tailored survey content based on demographic or behavioral data.
Example: “Test which intro message works better for different age groups.”
When to Use A/B Test Questions
Use A/B Test Questions when:
- You want to compare the effectiveness of multiple versions of a question or prompt.
- The goal is to refine survey design through data-backed testing.
- You need to assess how media, question wording, or layout impacts responses.
- Randomization or conditional logic-based distribution of variants is critical.
- You require a controlled test environment with built-in random assignment.
Scenario-Based Example
In a public health survey, researchers want to test two different phrasings of the same question about tobacco packaging. Participants are randomly assigned to one version:
Version A: “Based on the packaging shown, how harmful do you think this tobacco product would be if smoked daily?”
Version B: “How dangerous does the packaging make this product seem for daily use?”
Comparing responses reveals which framing more effectively communicates perceived risk.
Basic Setup
- Define Question Variants: Create two or more variations of the same question.
- Assign Answer Type: Choose from supported response options.
- Randomization: Ensure each respondent receives only one variant.
- Required Toggle: Make the question mandatory to proceed.
- Media Upload: Users can upload multiple media files to be randomized among participants.
Advanced Features
- Conditional Display Logic/Non-random Allocation: Show variants based on predefined set up condition.
Step-by-Step Guide
Step 1: Create a New Question
- Log into your Chisquares Investigator Portal.
- Navigate to the Survey Design section.
- Click Create a New Question.
Step 2: Select the Question Type
- In the question type section, select Group Allocation Questions.
- Choose Multiple Variant (A/B Test) from the list of available question types.
- Click Use Selected questionnaire item.
Step 3: Choosing a Foundational Question Type
- Click Change, and select one of the following foundational question types:
- Single Value Text
- Likert Scale
- Multiple Choice (MCQ)
- Sliding Scale
- Yes/No
- Multiple Responses
- Dropdown
- Prompts
Step 4: Add Your Question
-
Enter your question in the provided text box.
Example: Based on the packaging shown, how harmful do you think this tobacco product would be if smoked daily? -
Apply formatting using the toolbar.
-
Configure the foundational question settings as required.
-
The Multiple Variant (A/B Test) question inherits all properties of the foundational question type.
Step 5: Configure Variant Allocation
Multiple Variant (A/B Test) questions can be used for both random and non-random allocation of variants. By default, participants are assigned randomly. To switch to rules-based allocation:
- Set conditions for assigning variants.
- Tap 'Click here' to assign variants based on the set conditions.
- Adjust the distribution ratio if needed (e.g., 50/50 split or custom percentages).
Step 6: Make the Question Required (Optional)
If you want to require participants to answer this question, toggle the Required switch at the top-right of the panel. This ensures respondents cannot proceed without answering the question.
Step 7: Add Media (Optional)
You can upload media (images, videos, or audio files) to accompany the question. Here’s how:
- Click the paperclip icon at the top-right of the panel.
- Upload your media file (must be under 4MB).
- Once uploaded, choose the alignment for the media (left, right, or center). The default text overlay will say, “View the media below, then answer the following question.”
Preview Mode
Before finalizing the Multiple Variant (A/B Test) setup, it is important to preview how it will appear to participants.
- Click the Preview button to see the question as participants will view it.
- Make sure all options, media, and settings look as expected.
Final Step: Push to Questionnaire
Once you are satisfied with your Multiple Variant (A/B Test) setup:
- Click Push to Questionnaire to add the question to your live survey.
- If you need to make edits later, you can always return to the question editor to make changes.
Example Question Setup
Question
View the image below then answer the question that follows:Based on the packaging shown, how harmful do you think this tobacco product would be if smoked daily?
Responses
- Not harmful at all
- Slightly harmful
- Somewhat harmful
- Very harmful
- Extremely harmful
[Note: Each participant will see only one randomly assigned question along with any associated media uploaded by the researcher. Unlike randomized trials where different media are randomized to a single fixed question, here, multiple question variants are created, and each participant is randomized to see only one question.]
Additional Features:
- Required: No
- Randomized Variant Distribution: 50% each
- Change foundational question: Change the foundational question to an appropriate one for your Randomized Trial.
- Add Section Header (Optional): Toggle this switch to add a section header to the question for context or grouping.
- Add Footer Prompt (Optional): Toggle this switch to include a closing note or prompt at the bottom of the question.
Value Proposition on the Chisquares Platform
Chisquares provides a powerful, streamlined system for implementing A/B tests within surveys. With random and rules-based variant allocation, media integration, and full compatibility with foundational question types, researchers can conduct sophisticated testing with ease.
Key platform-specific advantages include:
-
Multi-Variant Support
Create two or more question variants, each with unique content, wording, or media. -
Random or Conditional Allocation
Distribute question variants randomly or based on custom logic (e.g., demographic filters). -
Compatible with Foundational Question Types
Works with:- Single Value Text
- Likert Scale
- Multiple Choice (MCQ)
- Sliding Scale
- Yes/No
- Multiple Responses
- Dropdown
- Prompts
-
Media Upload for Each Variant
Associate variant-specific images, videos, or audio clips to support testing goals. -
Custom Variant Distribution Ratios
Adjust response allocation (e.g., 50/50, 70/30) depending on sample size or test design. -
Edit and Preview Support
Visualize the user experience across all variants before deploying. -
Clean Data Export with Variant Labeling
Track which variant each respondent saw to segment and analyze test results.
Conclusion
The A/B Test question type in Chisquares is a powerful tool for evaluating different question formats and optimizing survey engagement. By leveraging randomized assignment and performance tracking, users can refine their content based on real-time respondent data. Whether testing product descriptions, marketing messages, or survey wording, A/B Testing ensures data-driven decision-making for superior results.
To watch a video of this question being set up, click the link below: link